This file with additional information is published using the packages rmarkdown (Allaire et al. 2020) and bookdown (Xie 2020). The package pivottabler (Bailiss 2021) was used to create the pivot tables.

We checked if SPSS and R would give comparable or identical results for the same Multilevel model if the hyperparameters were identically specified (all analysis used Maximun Likelihood, identical reference categories etc.). For simple models different packages almost always give identical results for the estimates for the fixed effects, random effects and other test statistics. Small differences are seen for the standard errors for the fixed effects, but they are usually small and do not influence the conclusions most of the times.

In this additional material, we only present the estimates and standard errors for the analysis of model 5 (see article) and for the analysis by country including the interaction effects of teacher gender and subject, number of students and experience. Results for the random effects are very similar to the presented results.

The data has been exported from SPSS into an EXCEL file and from R into a R Markdown HTML. The R Markdown was incorporated into the SPSS export and a new tab was created that was read into SPSS. These files are read into R Markdown. As the R output was limited to two decimals, all results are rounded to two decimals. The rounding errors were small and have no influence on the results shown in this file.

1 Clearing memory and loading packages for analyses

rm(list = ls())     # remove all variables, clean memory
curWD <- dirname(rstudioapi::getSourceEditorContext()$path) #Get the directory of current script
setwd(curWD)

library(pivottabler)

2 Analysis model 5 comparing estimates and standard errors between SPSS and R

casedata <- haven::read_spss("Model5_SPSS_R_Trans.sav")  

casedata$type <- factor(casedata$type, levels = c(1,2,3,4,5,6),
                   labels = c("GM", "teacher", "subject", "scales", "class", "country"))
pt <- PivotTable$new()
pt$addData(casedata)
pt$defineCalculation(calculationName="MeanFixedEstimate", caption="estimate fixed effect 
                     by (type of) parameter and software", 
                     summariseExpression="mean(FE_Estimate, na.rm=TRUE)", format = "%.2f")
pt$defineCalculation(calculationName="MeanFixedStdErr", caption="standard error fixed effect 
                     by (type of) parameter and software",  
                     summariseExpression="mean(FE_STDERR, na.rm=TRUE)", format = "%.2f")
pt$addColumnCalculationGroups()
pt$addColumnDataGroups("Software", addTotal=FALSE)
pt$addRowDataGroups("type", addTotal=FALSE)
pt$addRowDataGroups("Parameter", addTotal=FALSE)
pt$renderPivot()

3 Analysis by country including interactions comparing estimates and standard errors between SPSS and R

casedata <- haven::read_spss("Model5_Countries_SPSS_R_trans.sav")  

casedata$country <- factor(casedata$country, levels = c(1,2,3,4,5,6),
                   labels = c("Indonesia", "Mongolia", "Pakistan", "South Korea", "Spain", "the Netherlands"))

casedata$type <- factor(casedata$type, levels = c(1,2),
                   labels = c("main", "interaction"))

3.1 table parameters model 5

#estimates fixed effects by country analysis 
pt <- PivotTable$new()
pt$addData(casedata)
pt$defineCalculation(calculationName="MeanFixedEstimates", caption="esitmates fixed effect by country and software",
                     summariseExpression="mean(trans1, na.rm=TRUE)", format = "%.2f")
pt$addColumnCalculationGroups()
pt$addColumnDataGroups("country", addTotal=FALSE)
pt$addColumnDataGroups("software", addTotal=FALSE)
pt$addRowDataGroups("type", addTotal=FALSE)
pt$addRowDataGroups("predictors", addTotal=FALSE)
pt$renderPivot()
pt <- PivotTable$new()
pt$addData(casedata)
pt$defineCalculation(calculationName="MeanFixedStdErr", caption="standerd errors fixed effect 
                     by country and software", 
                     summariseExpression="mean(trans2, na.rm=TRUE)", format = "%.2f")
pt$addColumnCalculationGroups()
pt$addColumnDataGroups("country", addTotal=FALSE)
pt$addColumnDataGroups("software", addTotal=FALSE)
pt$addRowDataGroups("type", addTotal=FALSE)
pt$addRowDataGroups("predictors", addTotal=FALSE)
pt$renderPivot()

Literature

Allaire, J. J., Y. Xie, J. McPherson, J. Luraschi, K. Ushey, A. Atkins, H. Wickham, J. Cheng, W. Chang, and R. Iannone. 2020. Rmarkdown: Dynamic Documents for R (R Package Version 2.6). https://github.com/rstudio/rmarkdown.

Bailiss, Christopher. 2021. “Create Pivot Tables [R Package Pivottabler Version 1.5.2].” https://CRAN.R-project.org/package=pivottabler.

Xie, Y. 2020. Bookdown: Authoring Books and Technical Documents with R Markdown (R Package Version 0.21). https://github.com/rstudio/bookdown.